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ABSTBACT 

Foar case histories were studied in an on-going 
project to develop a nethod for user selection of parchased 
scientific and technical inforsation services. The issaes involved 
vere: (1) the valae of cospater search services to a ssaXl branch of 
a cospany technical library; (2) the special declsion-saking factors 
used for selecting itess of very high or lov costs; (3) the jodgesent 
factors applied ehen purchasing expensive special topic bulletins^ 
and (4) the differences in vork environsent and other variables 
between a university science library and an industrial research 
library* t survey of usage patterns for scientific and technical 
inforsation services at the Exxon Besearch and Engineering Cospany 
Central Library was planned. (PF) 
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A Generalized Method for the User Evetitttatiert 
Selection oE Purchased Informatio n S ervices ' 

Objectives and Deflnltloas 

The definition of the term services aa used herein has been 
considered further In the light of Its functional significance. 

A "simple searching service" Is one In wbich the question Is 
completely defined |>efore the search la started^ and any reatatemeat 
of the question represents a nec/ aearch. The distinction lies In hov 
the aervlce Is used^ not the mechanlca of the system. Redefinition of 
the question during the aearch requires a much higher degree of aklU 
and training on the part of the user. Such redefinitions are character- 
istics of Information analyals. In terms of the present contract Phase I 
is addressed to simple searching aervlces^ and Phase II will add the 
parameters for information analysis. 

This statement of the ^critical distinction between Phase I and - 
Phase II is tak^ln part from: the training principles for enginedra uaed 
by I^: statement of the problem to be solved Is called Input -to the 
system^ process operations are Internal to the system^ and output la 
tailored to meet predefined needa. Two guiding principles are recognized: 

a) the more sharply Input can be defined^ the easier 
It la to define an acceptable output^ and the less 
problem It la to satisfy the qualitative aspects 

of user needs; * ^ 

b) any Interaction between the customer and the syatem 
during processlaig changes the Input no matter^ hov 
allgjit the change. 

This creates a new statement of the problem^ and costa which may .be 
ruinous^ unless the changes are made by an expert who understands com-* 
pletely the language of the machine as veil as the language of the pro- 
blaa. This may save overall coats or It may not^ alnce It depends on 
Whether the time of s'ich an eKpert is available on call^ aa well as 
whether he can do the job in enough leaa time to more than balance his 
higher unit rate. 

This definition clarlfiea our concept of seinrlces to be 
included In Phase I^ since It says that When a compleic (computerized) 
service la being used for routine operation^ this may be a simple aearch^ 
even though the aame equipment can be used in aophiatlcated ways for 
information analysis. 
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The title of this report has been reworded on a trial basis 
as ^'a generalized method for the user selection of purchased services^" 
changing a word to selection Instead of evaluation* The term ^'evaluation 
of Information systems" has been used extensively In the literature^ but 
with different meanings* The most frequent meaning Is -apparently the 
evaluation of performance of an Individual system^ addressed to system 
design* Only a few references by comparison deal with the comparative 
evaluation of different systems^ which Is our problem^ for the user 
selection between competing services* This questions \tLll be considered 
further » and conmieats are specifically Invited* 

Case Histories Considered 

Four potential case histories were considered this months 
relating to the definition of S*T*I* service needs and cost factors lu 
special situations* Each of these Is being examined for factors In the 
selection of services: 

a) -A small branch or special, library ; ^. ■ - , 

JChe use;of mechanized or comjntterlzed searching seinrlces Is 
'frequently resisted by the small branch library^ which sees the new 
service as Incomplete^ regi^rdless 'c£ cost* A clear example appears 
In an exchange of letters in the Issue of Special Libraries for Movember 
l975f page 6A: the librarian at the-Harathon Oil Company research center 
(ll7 employees) points out the high base cost of data management 
systems for a small user» and an author from Chevron ^Research (over 
1^000 employees) replies that Incremental use by branch libraries can 
be cheap and very^useful^ once the base charge Is paid by a parent 
crganlzatlon* .„ ^The polemical .tone of both, writers makes it obvious that 
they, are 'not speaking the same language *^ 

- A similar example Is available tor study within Exxon: the 
situation can be examined as a problra in the definition of needs» in a 
system which is considering a basic change* There may be strong hidden 
factors Inv^olved* The small branch or departmental library may be in 
essence an expansion of the personal library of the department manager » 
created a±; a service center and not just an information source* It 
operates in a closed market^ which is not likely to be expanded regard- 
less of any Improvements* The special services it offers serve as a way 
of doing business^ not Just as a source of Information* The most valuable 
parts of the system may be a browsing file of collected documents^ 
abstracts or indexes^ the skill of the staff Indexer who creates thls» 
and the familiarity of the department personnel with how it works* If 
the skilled staff could be replaced by a moderate to low*cost incremental 
information package^ available from a parent organization^ this would 
deprive the branch operation of its chief reasons for separate existence* 
Also» the minimum times required for the training or skill maintenance 
of staff users of a sophisticated new system are a larger burden^ the 
smaller the operation* 
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In tbis situation^ the consideration of needs suggests that the 
best use of auxiliary information services inay be to upgrade the work 
input to the indexer; whose final selection and integration of new 
material into the files is one of the most valuable service the staff 
performs. The new information tools may be used to pre*6elect items 
to be considered for special indexing^ drawing either from the total 
flow of current source niaterial or as a secondary supplanent from the 
less productive sources^ which take more time to find pertinent items. 

An acceptable definition of needs may become how to pre^^select 
and feed to special inde:ctng the type of items which have proved most 
useful in current operations. This may be based as much on operative 
terms (such as types of interaction effects) as on topical dreas» or 
hierarchical relationships. Standard procedures are applicable. to the 
analysis of current experience^ to determine what specific files have 
been most powerful or most tn need of improvement. Other aspects of 
the decision matrix can be considered from this same point of view. 

b) Costs by range, not seriatim 

Comparisons between competing services are not usually based 
on cost aljone» but tempered to allow for di£ferences in the service 
offered. The assumption is that market forces keep directly competing 
services close to each other in price» and that the service which costs 
more has something more to offer - if this feature is one you want to 
buy. To the extent that this is true» cost becomes a secondary factor 
in the selection between services^ as long as they are within the same 
general range. This may be entirely contrary to the trade literature 
published by the vendors^ who are highly cost conscious and anxious 
to use this point as a selling tool. 

On this basiSy there would be only three significant ranges 
of cost: on the market^ too cheapo and too expensive. Differences 
within each range might be considered as relatively unimportant^ although 
varying over as much as one order of magnitude including the average. 
Prices far below the average are suspect^ because they suggest serious 
defects in the service offered. Prices within i^erhaps 50% o£ the average 
are assumed to correspond to desirable eittra features or allowable omis^ 
sionSy and may not be questioned seriously as long as the user has made 
up his mind as to what he wants. Prices above this range bring in new 
questions of their effect on total budgets^ of available alternates^ 
and of the certainty of use sufficient to justify special purchase. 
Adjustments are also necessary for the number of customers^ from a 
service which is cheap because of cost sharing at one end to a service 
which is expensive because it is proprietary* and distrli>ution to other 
customers is rigidly excluded. 

An example of this factor of costs by range is the decision of 
the central library at ER&E to cancel its standing order for continuing 
index volumes in series such as Beilstein* which are recognized as valuable 
tools for access to information. The problem is that Beilstein supple- 
ment IV has now reached the range of $400-500 per volume. Five ^ ^cn 
volumes appearing at unscheduled intervals in 1975 can ruin the book 
budget for the year^in even a fairly large library. Prices so far above 
the average force the consideration of alternative sources and procedures^ 
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no matter how us&ful the Item. In this case^ each purchase must be 
considered separately on Its merits^ and a possible anst^er is that other 
sources such as Chefmlcal Abstracts^ with computerized access^ may have to 
take the place of the rest of Bellsteln. 

The question of probable extent of use Interacts with costs 
somewhat differently In different ranges. For average costs» even within 
budget^ many libraries will Insist that a single user must make the pur- 
chase recommended with his own funds^ unless he can demonstrate that the 
service will enjoy multiple use by others as well. This requirement Is 
most likely to be waived for requests of low or average costs » In a new 
field of interest^ where there Is a presumption that others will become 
users after the new service Is made available. 

c) Special Topic Bulletins 

. The opening of any new field of scientific and technical Infor*- 
matlon is likely to bet greeted by one or more new. newsletters, or special 
bulletin services^ as soon as it can be-^Kpanded Into a recognizable 
market. These bulletins may be aimed p?:imarlly at the small user ^rho does ' 
not have his. own information service. This may be an Individual in 'even 
a large corporation; however » in a new field' where interest is limited^ 
or where the user .does not know how to ask foX information.^ Since these 
services are expensive and tend to become full of trivia^ special q^uallty 
factors mu4t be applied^ Including such criteria as. the following: 

■ qualitative ■ 

^ items come' from primary sources^ not just old ^materlal 
^ information cumulates^ access routes- supplied 
-. tied to known sources » 'improves access to 'other'niaterial 
suitable for browsing; organlzed'by related ideas 

judgmental ^ ' , . ; " . ^ ' 

- reputation of publisher probably a quality product 
importance to major company interests 

quantitative 

^ priced within the current market xenge 

^ number of users» sufficient to justify central purchaso 

d) A University Science library 

Differences in the work environment and other variables between 
a University and an industrial research laboratory have a significant 
effect on the parameters applied to the selection of purchased information 
services. The following list was suggested in a preliminary discussion 
with the Librarian of a nearby University Library of Science and Medicine: 

- The Library does no searching^ as a regular service^ 
for Itself or for other people^ but only provides 
tools for others to use. 
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- Seml-publlc use of the. facilities by unskilled users 
creates serious problems of mlsfilingv as compared to 
the Industrial library which enjoys a more protected 
environment and users of higher average skill* Thus» 
the I]hiversit;y Library refuses to purchase card 
services^ or loose-leaf with a steady flow of supple- 
; mentSy because they involve a block of time to get 
Into business^ interminable interi^iling thereafter^ 
and the constant hazard of items which are lost or 
misfiled* 

~ The Library is leary of highly specialised materials^ 
and requires a minimum number of users^ 'no mattj&r who 
they are* Any Individual re^^earch project or contract 
can be terminated^ and leave sn orphaned^ broken file* 
An index or access system which is needed by only one 
lAan must be purchased and maintained, by him» therefore^ 
out of project funds» regardless of how valuable it is 
to him* Such a file may be transferred to the library 
at a later date^ but only- after sufficient' use by 
others has been established by experience* ' ^ 

'Uniqueness of a given service on the market compared to 
its competitors comes before cost» In deciding which 
to keep and which to discard^ although both factors 
must be consideired* " 

^ Statistics on the use of different data bases are 

available for study from on-line billings for different 
user groups: f acuity » graduate students^ and uitder^' ^ 
graduates^ and by University Section for each* These 
can be examined to look for shifts nai patterns*'^ 

This discussion will be followed up In due course^ by an 
interview and an examination of the statistics available* 

Cost vs* Benefits ' 

Strong contributions to the rec^t literature on system evalua^ 
tlons have bera made In the field of air pollution control*' The data 
available here (or missing) have been studied Intensively In the effort 
to bslance the costs and prospective benefits of different alternatives* 
In a critical review of the "Systems Approach to Air Pollution Control/' 
Bibbero recognizes that the choice between control strategies can be 
approached by either cost^eff ectlveness or cost-^beneflt analyses: 
cost-effectiveness refers to the choice between all feasible alternatives 
as a least-cost strategy^ for reaching a predetermined goal» While cost*- 
beneflt attempts to rationalise the goals themselves^ by relating sn 
optimsl benefit level to the marginal costs of both benefits and con- 
trols* Cost/effectiveness techniques are In fairly routine use» but the 
effort to qusntlfy cost/benefit requires seemingly impossible inputs 
such as the dollar value of sunshine or aesthetic scenery (p 506) * As a 
result^ the conclusion has been itrated that cost/benefit evaluations are 
too complex to enforce (In sir pollution control) and too sophisticated ^ 
for our legal system (Bibbero p 14» Mandelker; Krler pp 207» 213 In Atklson) 
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A direct parallel appears between the three basic types of 
constraints recognized in a cost/benefit analysis for air pollution controls 
deemed possible or desirable over a 10-year tena^ and the dimensions of 
choice we have proposed for a matrix of factors in the selection of 
information services (p 202): 

System Constraints in 
Air Pollution Controls 



Social motivation 

reflected in legislation 

amounts and source ot power 
to be produced 

population^ transportation 

allocation of ^ costs 

Medical/health' interactiiond 
(continuing need for research) 

Cost/effectiveness analysis 
more nearly routine 



This is an instructive comparison: in both situ^itions there 
is an initial group of factors involving motivation or managment strategy 
which are hard to define but often decisive^ regardless of other variables. 
The wholiSi gamut. of costs and 'quantifiable factors may be'easler to 
handle^; by data or by analogy^ but they come at the other end of the 
selection process. Cost effectiveness alone carries little. wel^t unless 
the basic questions of need and liking receive affirmative answers. The 
statement that cost/benefit evaluations are too -complex to enforce 
legally is in line with this conclusion^ since **enf or cement*' implies 
an adversary system vhicl: inherently seeks to minimize factors of Indepen**. 
dent judgment. With even fragmentary data^^ the balance between increasing 
costs of controls and decreasing costs of effects (increasing benefits) 
with an increase in degree of control may be enough to compare present 
status with future possibilities and tell which direction to move^ in the 
search for further benefits. 

Current Strategy 

a) Survey of Usage Patterns 

Disparities between the evaluations which different users give 
to the same group of information services are familiar^ and disturbing: 
a first conclusion is that there are very few services on which all users 
would agree. Disagreements just as strong can be found in the comparative 
rankings assigned by a single user searching different questions. Thl3 
was indicated in the quarterly report^ Table 2. As the inverse of this 
it is easy to construct "test questions" which will intrinsically favor 
one service over another in a direct comparison^ (such as Ringdoc^ API 
Q Literature^ or Blosls). It Is the thesis of this project that these 
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Dimensions of Choice in 
Selection of STI Services 



>lanagement decision (motivation) 
is it our way of doing business 

amounts of information to he used» 
rather than doing work instead 

customers^ communication policy 

policy on costs 

Personal choice factors. ■ 
(subjective ranWlags) .; 

Quailti'fiable factors ' ' ^ ^ 

costs^ resources^ capabilities 



disagreements are not random^ and that they can be analyzed systematically 
to detect predictable patterns. 

Regular users of purchased information service at the £R&E 
Central Library include 15*20 people who spend an average of 50% or more 
of their working day in searching the literature. A third of these are 
library staffs half of thom information analysts or searchers who have 
special areas of interest^ and the others include scientists who are 
strongly oriented toward the use of the literature. All of these groups 
comprise individuals with different levels of training^ slcill» and res- 
ponsibility - including the freedom to decide which searches are most 
worth vhile» and what information will actually be tised. Preliminary 
information developed to date» including conclusions front Tables 1 and 2 ' 
in the quarterly report^ has led to the postulate that certain types of 
usersy in cex^taln situations^ will select certain types of services. 

A survey of usage patterns for these regular users is being, 
organized » to provide a data base and check points to confirm or change 
these postulates and conclusions. The present plan is to conduct this 
survey for a months after completing plans during December. The first 
step in the survey is to conduct an Initial interview with. major users» 
based on their records .of past experience^ to -determine ^at Inf ormation 
services each one uses most and about how often. This may be extended 
to the whole group. The results will determine what questiond are worth 
asking » and what services^should be added or removed from the previous 
lists (Tables 1 and 2)^ which were based on the opinions of four people. 
Present indications are that many users ma^ havfe one or two favorites^ 
which this individual uses more than all other* services put together 
(CA» I^e^^nty company reports file) . If this is correct^ the survey for 
each individual may be greatly simplified by identifying this pattern^ 
getting enough data to confirm ±t, in an initial few days» and concentrating 
thereafter on Incremental uses of other services. The approach being 
considered^ subject to confirmation in further initial .interviews^' could ^ 
be to ask each user to identify by a hash mark on a tally sheet each use . 
of the services selected (except possibly those used most often and , , 
already documented) » with a special marking for those uses found most 
valuable in a given search. These sheets can be collected and analyzed ^ 
dally» or at a suitable interval^ and serve as the basis for follov-^p 
interviews , to check out items of special interest. It is desirable to 
use an approach which Involves an absolute minimum of record keeping^ 
to elicit the continued cooperation of a large number of persons'. The 
frequent collection of data and regular follow-up by phone or by visits 
for items selected because they represent success^ will serve as a 
stimulus for continued cooperation. Various parts of this survey can 
be checked against a variety of previous internal surveys of patterns 
of use. 



b) Literature Review 



The literature available and collected to date has been studied 
primarily to Consider the basic phllosphy of the selection matrix pro- 
posedy and the semantic problecus in the definition of key terms such as 
**user** or "service." The term "evaluation". has as many aspects as others: 
It Is frequently addressed to the problem of hoir to improve the opera-^ 
tlon of a single system rather than hour to compare It with others^ which 
Is the object of this program. 

It Is time to return to the literature^ , as these concepts becoma 
clarified In our own thinkings to look for additional Input. We are 
particularly Interested In data which provide comparisons In simple 
searching^ of a pre-defined question^ which may employ complex tools 
but which does not penult re*deflnltlon of the question during the search- 
ing process. We consider this definition of simple searching as the 
critical distinction between the present Fhase X of the two year project^ 
and the Information analysts type of operation to be covered in Phase II- 

1 t* * * mm m ' ' ' ■ ' 

LI * 

' Our literature review will be directed at specific data» showing 
comparisoiis between different seicvlced^' and Incraoental references per- 
tinent to patterns or dimensions in the selection matrix proposed. . 
Additional personnel for this review has been provided by Dr. Klrshenba^jm^ 
as 'previously discussed with you. Our searcher will be Dr. Walter Herbst^ 
a consultant Senior Research Associate^ recently retired^ who has been 
working with Dr. Kirshenbaum as an information analyst. Background 
information has been supplied to him» and he will work with us part time» 
starting in December^ for a period to be determined by what he finds. 

^ ., ' * ■ 

c) Special 'interviews " ■ 

A "unique" service^ one for which there Is no substitute^ Is 
not neceasarily limited to a large and expensive operation. Since this 
category has been chosen for special study» It is useful to examine the 
factors which apply to the users of a "unique" system which is very 
smally compared to the major indexing/abstranCting services listed in ^ 
Table 1 and our proposed usage aurvey. 

Preliminary discussions have been held with the Director of 
"Accent on Information^" a special service of **Accent on Living" for 
handicapped individuals. This service answers their questions regarding 
technical procedures^ equlpmentj^ or background data available for 
physical^ economic and social needs. In such a small gytem» as In 
others^ one key to success Is "question negotiation" before the start 
of a search^ to translate It from user language to system language. 
User motivation Is exceedingly high; this has^ been capitalized on to 
upgrade the operation by publicizing to the users any question which the 
system could not answer from its files. The result has been 100% 
answers provided, by data or by the closet analogy available. This feed- 
back is augmented by an In-'depth study and special monographs for most- 
frequent questions. Technical information inonographs In this series 
have Included the design of kitchens to be used from ^ whell-chalr» 
and the sexual problems of paraplegics. These titles have regularly 
sold several thousand copies within 6 weeks» amoog a total membership 
of 15»000» because diere 1$ no other source for such information. 
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Factors which apply to this type of **uniqu& service** include 
the concepts of reputation^ factuality^ feedback and responsiveness^ which 
are quite similar but not always identical to the way these factors apply 
to larger systems. Furdicr discussions will be held to clarify any dif^ 
ferences or apparent shifts in emphasis due to changing parameters of 
size^ user acceptance^ or others which can be identified. 

A follow-up conversation has been held with Mr. Ben luberoff^ 
Editor of the ACS Journal ChenAec^ regarding a possible general interest 
article in his journal based on our Interim Report* He is favorably 
inclined at this time^ and is tentatively reserving space for a mid"* 
susnmer article^ possibly July^ based on text which we ccnild supply as 
a first draft in February* His readership includes many of the research 
managers or eicecutives who are decision-^makers for the purchase of 
information services^ and a very desirable audience for us to reach. 
The timing proposed to him and tentatively accepted would put the 
article in print at a time which is good to direct attention to our 
forthcoming, final report^ to be available as of July/August or shortly 
thereafter* 

This arrangement does not constitute a definite commitment 
either on our part as author or on his part as publisher^ but it is 
necessary for it to be outlined and agreed upon xwt if it is to be 
available as an option when we want it. 

* . *^ * 

Respectfully £;ubmitted^ 

4looter J. Hall '. 
Principal Investigator ' 
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